{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Inference TensorFlow Bert Model with ONNX Runtime on CPU"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "In this tutorial, you'll be introduced to how to load a Bert model using TensorFlow, convert it to ONNX using tf2onnx, and inference it for high performance using ONNX Runtime. In the following sections, we are going to use the Bert model trained with Stanford Question Answering Dataset (SQuAD) dataset as an example. Bert SQuAD model is used in question answering scenarios, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 0. Prerequisites ##\n",
    "First we need a python environment before running this notebook.\n",
    "\n",
    "You can install [AnaConda](https://www.anaconda.com/distribution/) and [Git](https://git-scm.com/downloads) and open an AnaConda console when it is done. Then you can run the following commands to create a conda environment named cpu_env:\n",
    "\n",
    "```console\n",
    "conda create -n cpu_env python=3.8\n",
    "conda activate cpu_env\n",
    "conda install -c anaconda ipykernel\n",
    "conda install -c conda-forge ipywidgets\n",
    "python -m ipykernel install --user --name=cpu_env\n",
    "```\n",
    "\n",
    "Finally, launch Jupyter Notebook and you can choose cpu_env as kernel to run this notebook.\n",
    "\n",
    "Let's install [Tensorflow](https://www.tensorflow.org/install), [OnnxRuntime](https://microsoft.github.io/onnxruntime/), [tf2onnx](https://github.com/onnx/tensorflow-onnx) and other packages like the following:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import sys\n",
    " \n",
    "!{sys.executable} -m pip install --quiet --upgrade tensorflow==2.6.0\n",
    "!{sys.executable} -m pip install -i https://test.pypi.org/simple/ ort-nightly\n",
    "!{sys.executable} -m pip install --quiet --upgrade tf2onnx==1.9.2\n",
    "!{sys.executable} -m pip install --quiet transformers==4.9.2\n",
    "!{sys.executable} -m pip install --quiet onnxconverter_common\n",
    "!{sys.executable} -m pip install --quiet psutil wget pandas"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Looking in indexes: https://test.pypi.org/simple/\n",
      "Requirement already satisfied: ort-nightly in /bert_ort/wy/anaconda3/envs/cpu_env/lib/python3.8/site-packages (1.8.2.dev20210901001)\n",
      "Requirement already satisfied: protobuf in /bert_ort/wy/anaconda3/envs/cpu_env/lib/python3.8/site-packages (from ort-nightly) (3.17.3)\n",
      "Requirement already satisfied: numpy>=1.16.6 in /bert_ort/wy/anaconda3/envs/cpu_env/lib/python3.8/site-packages (from ort-nightly) (1.19.5)\n",
      "Requirement already satisfied: flatbuffers in /bert_ort/wy/anaconda3/envs/cpu_env/lib/python3.8/site-packages (from ort-nightly) (1.12)\n",
      "Requirement already satisfied: six>=1.9 in /bert_ort/wy/anaconda3/envs/cpu_env/lib/python3.8/site-packages (from protobuf->ort-nightly) (1.15.0)\n"
     ]
    }
   ],
   "metadata": {
    "scrolled": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's define some constants:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# Whether allow overwrite existing script or model.\n",
    "enable_overwrite = False\n",
    "\n",
    "# Number of runs to get average latency.\n",
    "total_runs = 100\n",
    "\n",
    "# Max sequence length for the export model\n",
    "max_sequence_length = 512"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "import os\n",
    "cache_dir = './cache_models'\n",
    "output_dir = './onnx_models'\n",
    "for directory in [cache_dir, output_dir]:\n",
    "    if not os.path.exists(directory):\n",
    "        os.makedirs(directory)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "import tensorflow as tf\n",
    "tf.config.set_visible_devices([], 'GPU') # Disable GPU for fair comparison"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 1. Load Pretrained Bert model ##"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Start to load fine-tuned model. This step take a few minutes to download the model for the first time."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "from transformers import (TFBertForQuestionAnswering, BertTokenizer)\n",
    "\n",
    "model_name_or_path = 'bert-large-uncased-whole-word-masking-finetuned-squad'\n",
    "#model_name_or_path = \"bert-base-cased\"\n",
    "is_fine_tuned = (model_name_or_path == 'bert-large-uncased-whole-word-masking-finetuned-squad')\n",
    "\n",
    "# Load model and tokenizer\n",
    "tokenizer = BertTokenizer.from_pretrained(model_name_or_path, do_lower_case=True, cache_dir=cache_dir)\n",
    "model = TFBertForQuestionAnswering.from_pretrained(model_name_or_path, cache_dir=cache_dir)\n",
    "# Needed this to export onnx model with multiple inputs with TF 2.2\n",
    "model._saved_model_inputs_spec = None"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "All model checkpoint layers were used when initializing TFBertForQuestionAnswering.\n",
      "\n",
      "All the layers of TFBertForQuestionAnswering were initialized from the model checkpoint at bert-large-uncased-whole-word-masking-finetuned-squad.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForQuestionAnswering for predictions without further training.\n"
     ]
    }
   ],
   "metadata": {
    "scrolled": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 2. TensorFlow Inference\n",
    "\n",
    "Use one example to run inference using TensorFlow as baseline."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "import numpy\n",
    "\n",
    "question, text = \"What is ONNX Runtime?\", \"ONNX Runtime is a performance-focused inference engine for ONNX models.\"\n",
    "# Pad to max length is needed. Otherwise, position embedding might be truncated by constant folding.\n",
    "inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors='tf',\n",
    "                               max_length=max_sequence_length, pad_to_max_length=True, truncation=True)\n",
    "output = model(inputs)\n",
    "start_scores, end_scores = output.start_logits, output.end_logits\n",
    "\n",
    "num_tokens = len(inputs[\"input_ids\"][0])\n",
    "if is_fine_tuned:\n",
    "    all_tokens = tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0])\n",
    "    print(\"The answer is:\", ' '.join(all_tokens[numpy.argmax(start_scores) : numpy.argmax(end_scores)+1]))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/bert_ort/wy/anaconda3/envs/cpu_env/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2184: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "The answer is: a performance - focused inference engine for on ##nx models\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "import time\n",
    "start = time.time()\n",
    "for _ in range(total_runs):\n",
    "    outputs = model(inputs)\n",
    "end = time.time()\n",
    "print(\"Tensorflow Inference time for sequence length {} = {} ms\".format(num_tokens, format((end - start) * 1000 / total_runs, '.2f')))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Tensorflow Inference time for sequence length 512 = 1360.38 ms\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3. Export model to ONNX using tf2onnx\r\n",
    "\r\n",
    "Now we use tf2onnx to export the model to ONNX format.\r\n",
    "Note that we could also convert tensorflow checkpoints to pytorch(supported by huggingface team, ref:https://huggingface.co/transformers/converting_tensorflow_models.html) and then convert to onnx using torch.onnx.export()."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "import tf2onnx\n",
    "tf2onnx.logging.set_level(tf2onnx.logging.ERROR)\n",
    "\n",
    "output_model_path =  os.path.join(output_dir, 'tf2onnx_{}.onnx'.format(model_name_or_path))\n",
    "opset_version = 13\n",
    "use_external_data_format = False\n",
    "\n",
    "specs = []\n",
    "for name, value in inputs.items():\n",
    "    dims = [None] * len(value.shape)\n",
    "    specs.append(tf.TensorSpec(tuple(dims), value.dtype, name=name))\n",
    "\n",
    "if enable_overwrite or not os.path.exists(output_model_path):\n",
    "    start = time.time()\n",
    "    _, _ = tf2onnx.convert.from_keras(model,\n",
    "                                      input_signature=tuple(specs),\n",
    "                                      opset=opset_version,\n",
    "                                      large_model=use_external_data_format,\n",
    "                                      output_path=output_model_path)\n",
    "    print(\"tf2onnx run time = {} s\".format(format(time.time() - start, '.2f')))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 4. Inference the Exported Model with ONNX Runtime"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we are ready to inference the model with ONNX Runtime."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "import psutil\n",
    "import onnxruntime\n",
    "import numpy\n",
    "\n",
    "sess_options = onnxruntime.SessionOptions()\n",
    "\n",
    "# Set the intra_op_num_threads\n",
    "sess_options.intra_op_num_threads = psutil.cpu_count(logical=True)\n",
    "\n",
    "# Providers is optional. Only needed when you use onnxruntime-gpu for CPU inference.\n",
    "session = onnxruntime.InferenceSession(output_model_path, sess_options, providers=['CPUExecutionProvider'])\n",
    "\n",
    "batch_size = 1\n",
    "inputs_onnx = {k_: numpy.repeat(v_, batch_size, axis=0) for k_, v_ in inputs.items()}\n",
    "\n",
    "# Warm up with one run.\n",
    "results = session.run(None, inputs_onnx)\n",
    "\n",
    "# Measure the latency.\n",
    "start = time.time()\n",
    "for _ in range(total_runs):\n",
    "    results = session.run(None, inputs_onnx)\n",
    "end = time.time()\n",
    "print(\"ONNX Runtime cpu inference time for sequence length {} (model not optimized): {} ms\".format(num_tokens, format((end - start) * 1000 / total_runs, '.2f')))\n",
    "del session"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "ONNX Runtime cpu inference time for sequence length 512 (model not optimized): 1185.30 ms\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "# Some weights of TFBertForQuestionAnswering might not be initialized without fine-tuning.\n",
    "if is_fine_tuned:\n",
    "    print(\"***** Verifying correctness (TensorFlow and ONNX Runtime) *****\")\n",
    "    print('start_scores are close:', numpy.allclose(results[1], start_scores.cpu(), rtol=1e-05, atol=1e-04))\n",
    "    print('end_scores are close:', numpy.allclose(results[0], end_scores.cpu(), rtol=1e-05, atol=1e-04))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "***** Verifying correctness (TensorFlow and ONNX Runtime) *****\n",
      "WARNING:tensorflow:From <ipython-input-10-1948b56cd554>:4: _EagerTensorBase.cpu (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.identity instead.\n",
      "start_scores are close: True\n",
      "end_scores are close: True\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 5. Model Optimization\n",
    "\n",
    "[ONNX Runtime BERT Model Optimization Tools](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers) is a set of tools for optimizing and testing BERT models. Let's try some of them on the exported models."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### BERT Optimization Script\n",
    "\n",
    "The script **optimizer.py** can help optimize BERT model exported by PyTorch, tf2onnx or keras2onnx. Since our model is exported by tf2onnx, we shall use **--model_type bert_tf** parameter.\n",
    "\n",
    "It will also tell whether the model is fully optimized or not. If not, that means you might need change the script to fuse some new pattern of subgraph."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "!{sys.executable} -m pip install --quiet coloredlogs sympy \n",
    "\n",
    "optimized_model_path =  os.path.join(output_dir, 'tf2onnx_{}_opt_cpu.onnx'.format(model_name_or_path))\n",
    "\n",
    "from onnxruntime.transformers import optimizer\n",
    "optimized_model = optimizer.optimize_model(output_model_path, model_type='bert_tf', num_heads=12, hidden_size=768)\n",
    "optimized_model.use_dynamic_axes()\n",
    "optimized_model.save_model_to_file(optimized_model_path)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "failed in shape inference <class 'AttributeError'>\n",
      "failed in shape inference <class 'AttributeError'>\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We run the optimized model using same inputs. The inference latency might be reduced after optimization. The output result is the same as the one before optimization."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "session = onnxruntime.InferenceSession(optimized_model_path, sess_options)\n",
    "# use one run to warm up a session\n",
    "session.run(None, inputs_onnx)\n",
    "\n",
    "# measure the latency.\n",
    "start = time.time()\n",
    "for _ in range(total_runs):\n",
    "    opt_results = session.run(None, inputs_onnx)\n",
    "end = time.time()\n",
    "print(\"ONNX Runtime cpu inference time on optimized model: {} ms\".format(format((end - start) * 1000 / total_runs, '.2f')))\n",
    "del session"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "ONNX Runtime cpu inference time on optimized model: 1076.50 ms\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "print(\"***** Verifying correctness (before and after optimization) *****\")\n",
    "print('start_scores are close:', numpy.allclose(opt_results[0], results[0], rtol=1e-05, atol=1e-04))\n",
    "print('end_scores are close:', numpy.allclose(opt_results[1], results[1], rtol=1e-05, atol=1e-04))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "***** Verifying correctness (before and after optimization) *****\n",
      "start_scores are close: True\n",
      "end_scores are close: True\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Model Results Comparison Tool\n",
    "\n",
    "If your BERT model has three inputs, a script compare_bert_results.py can be used to do a quick verification. The tool will generate some fake input data, and compare results from both the original and optimized models. If outputs are all close, it is safe to use the optimized model.\n",
    "\n",
    "Example of comparing the models before and after optimization:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "# The baseline model is exported using max sequence length, and no dynamic axes\n",
    "!{sys.executable} -m onnxruntime.transformers.compare_bert_results --baseline_model $output_model_path --optimized_model $optimized_model_path --batch_size 1 --sequence_length $max_sequence_length --samples 10"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "100% passed for 10 random inputs given thresholds (rtol=0.001, atol=0.0001).\n",
      "maximum absolute difference=0\n",
      "maximum relative difference=0\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Performance Test Tool\n",
    "\n",
    "This tool measures performance of BERT model inference using OnnxRuntime Python API.\n",
    "\n",
    "The following command will create 100 samples of batch_size 1 and sequence length 128 to run inference, then calculate performance numbers like average latency and throughput etc."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "THREAD_SETTING = '-n {}'.format(psutil.cpu_count(logical=True))\n",
    "\n",
    "!{sys.executable} -m onnxruntime.transformers.bert_perf_test --model $optimized_model_path --batch_size 1 --sequence_length 128 --samples 100 --test_times 1 $THREAD_SETTING"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=100, test_times=1, use_gpu=False, intra_op_num_threads=24, seed=3, verbose=False)\n",
      "Generating 100 samples for batch_size=1 sequence_length=128\n",
      "Running test: model=tf2onnx_bert-large-uncased-whole-word-masking-finetuned-squad_opt_cpu.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=24,batch_size=1,sequence_length=128,test_cases=100,test_times=1,use_gpu=False\n",
      "Average latency = 363.76 ms, Throughput = 2.75 QPS\n",
      "Test summary is saved to onnx_models/perf_results_CPU_B1_S128_20210901-191653.txt\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's load the summary file and take a look. In this machine, the best result is achieved by OpenMP. The best setting might be difference using different hardware or model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "import glob     \n",
    "import pandas\n",
    "\n",
    "latest_result_file = max(glob.glob(os.path.join(output_dir, \"perf_results_*.txt\")), key=os.path.getmtime)\n",
    "result_data = pandas.read_table(latest_result_file)\n",
    "print(latest_result_file)\n",
    "\n",
    "result_data.drop(['model', 'graph_optimization_level', 'batch_size', 'sequence_length', 'test_cases', 'test_times', 'use_gpu'], axis=1, inplace=True)\n",
    "result_data.drop(['Latency_P50', 'Latency_P75', 'Latency_P90', 'Latency_P95'], axis=1, inplace=True)\n",
    "cols = result_data.columns.tolist()\n",
    "cols = cols[-4:] + cols[:-4]\n",
    "result_data = result_data[cols]\n",
    "result_data"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "./onnx_models/perf_results_CPU_B1_S128_20210901-191653.txt\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   Latency(ms)  Latency_P99  Throughput(QPS)  intra_op_num_threads\n",
       "0       363.76       550.04             2.75                    24"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latency(ms)</th>\n",
       "      <th>Latency_P99</th>\n",
       "      <th>Throughput(QPS)</th>\n",
       "      <th>intra_op_num_threads</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>363.76</td>\n",
       "      <td>550.04</td>\n",
       "      <td>2.75</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 6. Additional Info\n",
    "\n",
    "Note that running Jupyter Notebook has impact on performance result since Jupyter Notebook is using system resources like CPU and memory etc. It is recommended to close Jupyter Notebook and other applications, then run the performance test tool in a console to get more accurate performance numbers.\n",
    "\n",
    "We have a [benchmark script](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/run_benchmark.sh). It is recommended to use it to measure inference speed of OnnxRuntime.\n",
    "\n",
    "[OnnxRuntime C API](https://github.com/microsoft/onnxruntime/blob/main/docs/C_API.md) could get slightly better performance than python API. If you use C API in inference, you can use OnnxRuntime_Perf_Test.exe built from source to measure performance instead.\n",
    "\n",
    "Here is the machine configuration that generated the above results. The machine has GPU but not used in CPU inference.\n",
    "You might get slower or faster result based on your hardware."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "!{sys.executable} -m pip install --quiet py-cpuinfo py3nvml\n",
    "!{sys.executable} -m onnxruntime.transformers.machine_info --silent"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{\n",
      "  \"gpu\": {\n",
      "    \"driver_version\": \"455.45.01\",\n",
      "    \"devices\": [\n",
      "      {\n",
      "        \"memory_total\": 16945512448,\n",
      "        \"memory_available\": 16941318144,\n",
      "        \"name\": \"Tesla V100-PCIE-16GB\"\n",
      "      },\n",
      "      {\n",
      "        \"memory_total\": 16945512448,\n",
      "        \"memory_available\": 16941318144,\n",
      "        \"name\": \"Tesla V100-PCIE-16GB\"\n",
      "      },\n",
      "      {\n",
      "        \"memory_total\": 16945512448,\n",
      "        \"memory_available\": 16941318144,\n",
      "        \"name\": \"Tesla V100-PCIE-16GB\"\n",
      "      },\n",
      "      {\n",
      "        \"memory_total\": 16945512448,\n",
      "        \"memory_available\": 5449973760,\n",
      "        \"name\": \"Tesla V100-PCIE-16GB\"\n",
      "      }\n",
      "    ]\n",
      "  },\n",
      "  \"cpu\": {\n",
      "    \"brand\": \"Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz\",\n",
      "    \"cores\": 24,\n",
      "    \"logical_cores\": 24,\n",
      "    \"hz\": \"2593997000,0\",\n",
      "    \"l2_cache\": 262144,\n",
      "    \"flags\": \"3dnowprefetch,abm,adx,aes,apic,avx,avx2,bmi1,bmi2,clflush,cmov,constant_tsc,cpuid,cx16,cx8,de,erms,f16c,fma,fpu,fsgsbase,fxsr,hle,ht,hypervisor,invpcid,invpcid_single,lahf_lm,lm,mca,mce,md_clear,mmx,movbe,msr,mtrr,nopl,nx,osxsave,pae,pat,pcid,pclmulqdq,pdpe1gb,pge,pni,popcnt,pse,pse36,pti,rdrand,rdrnd,rdseed,rdtscp,rep_good,rtm,sep,smap,smep,ss,sse,sse2,sse4_1,sse4_2,ssse3,syscall,tsc,vme,xsave,xsaveopt,xtopology\",\n",
      "    \"processor\": \"x86_64\"\n",
      "  },\n",
      "  \"memory\": {\n",
      "    \"total\": 473403277312,\n",
      "    \"available\": 413617876992\n",
      "  },\n",
      "  \"os\": \"Linux-4.15.0-1103-azure-x86_64-with-glibc2.17\",\n",
      "  \"python\": \"3.8.11.final.0 (64 bit)\",\n",
      "  \"packages\": {\n",
      "    \"transformers\": \"4.9.2\",\n",
      "    \"torch\": \"1.9.0\",\n",
      "    \"tensorflow\": \"2.6.0\",\n",
      "    \"sympy\": \"1.8\",\n",
      "    \"protobuf\": \"3.17.3\",\n",
      "    \"ort-nightly\": \"1.8.2.dev20210901001\",\n",
      "    \"onnxconverter-common\": \"1.8.1\",\n",
      "    \"onnx\": \"1.10.1\",\n",
      "    \"numpy\": \"1.19.5\",\n",
      "    \"flatbuffers\": \"1.12\"\n",
      "  },\n",
      "  \"onnxruntime\": {\n",
      "    \"version\": \"1.8.2\",\n",
      "    \"support_gpu\": false\n",
      "  },\n",
      "  \"pytorch\": {\n",
      "    \"version\": \"1.9.0+cu102\",\n",
      "    \"support_gpu\": true,\n",
      "    \"cuda\": \"10.2\"\n",
      "  },\n",
      "  \"tensorflow\": {\n",
      "    \"version\": \"2.6.0\",\n",
      "    \"git_version\": \"v2.6.0-rc2-32-g919f693420e\",\n",
      "    \"support_gpu\": true\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.11 64-bit ('cpu_env': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  },
  "interpreter": {
   "hash": "074a0f2de953c1a97ea00d0646e5f1fda38562dc7d977159c4c016157763bc64"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
